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Dear Sir 

APPELLANTS' REPLY BRIEF 



The scope of this reply 

This brief responds to the Examiner's Answer mailed on December 7, 2005. In 
Section 9 at pages 3-12 of that Answer, the Examiner repeated the grounds for rejection 
as stated in the final rejection mailed April 26, 2005. Since the basis for the final 
rejection has already been addressed in appellants' Appeal Brief, the following comments 
are directed to the statements made in Section 10, the "Response to Argument," which 
appears on pages 12-15 of the Examiner's Answer. 

Nelson does not suggest the claimed invention 

The present invention uses conventional text-based Internet search engines to 
index and search the content of Web pages and the multimedia content, such as image, 
audio and video files, incorporated into those web pages by reference. As claimed, a 
Web page to be indexed is processed to identify the media data incorporated by reference 
into that Web page and to extract information describing the media data thus identified 
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from the referencing Web page, from the media data itself, and from other sources. The 
extracted information is then inserted as text annotations into a copy of the original Web 
page to produce an enhanced Web page which may then be processed in the normal way 
by conventional text-based Internet indexing and searching facilities. 

Nelson neither discloses nor suggests the invention. While the Examiner 
concedes in his Answer that Nelson does not expressly teach combining annotations with 
the original Web page to form an enhanced Web page as detailed in independent claims 
1, 8 and 10, he nevertheless maintains his contention that Nelson "suggests" the subject 
matter claimed. Appellants submit that there is no such suggestion anywhere in the 
Nelson patent. 

Nelson, like the appellants, describes a system for indexing and searching Web 
pages which contain references to external multimedia resources, such as image, audio 
and video files. In both systems, the Web pages are analyzed to identify the external 
multimedia objects, and then each such object is processed to generate metadata (tokens) 
describing the content of that external object. In both systems, this metadata may consist 
of character text, such as the words recognized by speech recognition processing of an 
audio file, or the words found in the close caption text of a video file. 

But appellants' invention as claimed differs from Nelson's system in the way this 
metadata, once extracted from the multimedia files, are used to index and search for Web 
pages and referenced multimedia content. 

As claimed, appellants' invention combines the extracted metadata with the 
original Web page to form an enhanced, annotated Web page that can then thereafter be 
indexed and searched in the normal way by conventional Internet search engines ("Web 
crawlers" or "spiders") such as those publicly available at www. hotbot. com, 
www. al tavista. com, www.excite.com. www.lvcos.com. etc. 

But Nelson does not insert the extracted data into Web pages to form enhanced 
Web pages, and nowhere "suggests" doing so. Instead, Nelson instead sorts and 
processes the tokens to form an inverted file in which each token is stored with data 
identifying the positions in the Web pages where each reference to a multimedia object 
described by that token appeared. 
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Contrary to the Examiner's assertion, one skilled in the art, reading Nelson, would 
not be motivated to use appellants* invention instead of Nelson's indexing system 
because, among other reasons, to do would destroy the workability of important functions 
which Nelson performs, including the ability to use the unified multimedia inverted index 
to perform numerous queries which conventional search engines cannot perform. For 
example, as Nelson notes at column 3, lines 6-18: "a compound query can include text, 
image, audio, or video components, and search operators that define . . . proximity 
relationships (e.g. "NEAR", "within n") between the components. For example, a 
compound query may require that the word "beach" appear within 10 words of an image 
of a sunset. The present invention uses the position information in the multimedia index, 
along with the indexed data (e.g. color data) to find compound documents that have the 
desired text and have an image with image characteristics (e.g. color, texture, luminance) 
which match the input query image within the defined proximity relationship. " If 
Nelson's system were modified to place the extracted tokens as text meta tags in the 
original Web page, instead of placing the tokens into the unified index along with 
positional information and other associated data, such as color data,, such special query 
functions could not be performed. 

Alternatively, if Nelson's inverted file indexing system was retained as taught in 
order to preserve its desired capabilities, it would serve no useful purpose to also insert 
the tokens into the Web page from which they can then be extracted to perform indexing, 
since those tokens are already available and placing them in the Web page would merely 
require additional processing and consume valuable Web page storage space without 
serving any useful purpose. 

Specific arguments advanced by the Examiner 

The Examiner notes that Nelson teaches that his system can be used to index 
HTML documents (Web pages), that the HTML standard teaches that text annotation 
(meta tags) can be included in a Web page, that conventional search indexes can use 
these meta tags to classify the document, and that Nelson thus "has knowledge of the 
advantages of imbedding character-based text annotations directly into the document to 
simplify the indexing of the document." From these observations, the Examiner 
then leaps to the conclusion that: 
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"Therefore, the Examiner maintains the belief that Nelson suggests, via 
knowledge of HTML documents, to one of ordinary skill in the art at the time of the 
invention that it was advantageous to have imbedded character-based text 
annotations directly into the document itself " 

But Nelson suggests no such thing. Nelson teaches a different way of indexing 
documents and nowhere "suggests * * * that it was advantageous to have imbedded 
character-based text documents directly into the document itself" 

The Examiner then goes on to say that one of ordinary skill would have: 

" . . considered modifying Nelson by imbedding rather than linking the 
character-based tokens into the multimedia documents, thus containing the 
metadata annotations within the document as is taught by the meta element of 
HTML. By embedding all of the tokens into the document, all of the meta data 
would have been extractable form a single source, the parent document, - thus 
simplifying the software extraction of the character-based text annotations for 
indexing the multimedia document 
This argument repeats a contention made in rejection mailed on January 31, 2005 that 
was dropped from the final rejection and now reappears in the Examiner's Answer. The 
contention is dealt with on page 7 of the Appeal Brief, and need be only briefly addressed 
here. One skilled in the art would not think that embedding the tokens in the document 
would "simplify the software extraction"'' as the Examiner claims. The tokens need to be 
extracted from the multimedia data in both systems. One skilled in the art would have no 
reason to modify Nelson's system to insert the tokens into a Web page and then take 
them out again. That modification would plainly complicate, not simplify. Nelson's 
system. 

The Examiner suggests further that: 

"Such a modification to Nelson would not render it unsatisfactory for its intended 
purpose because the intended purpose of Nelson is to provide a unified index of the 
multimedia document and all of its multimedia components and this is performed 
by the creation and indexing of the character-based tokens describing the content 
of the multimedia document and each of its multimedia components. Nelson fulfills 
its purpose of providing a unified index regardless of whether the character-based 



PAGE 5/11 * RCVD AT 2/6/2006 12:53:40 PM [Eastern Standard Time] * SVR:USPTO-EFXRF-6/35 * DNIS:2738300 * CSID: 1-508-620-6540 CCALL * DURATION <mm-ss):06-48 



To: U.S. Patent and Trademark Office Page 6 of 1 1 



2006-02-06 17:53:58 (GMT) 



1-508-629-6540 CCALL From: Charles Call 



tokens are associative]}? linked to the document, or imbedded within the 
document The indexing of the tokens would be the same regardless of where they 
are stored and Nelson would successfully commonly index the multimedia 
documents and their multimedia components either way. 
This contention ignores how Nelson's indexing scheme actually works. Nelson processes 
Web pages by first analyzing each Web page to identify references to external 
multimedia components (see Figs. 4 and 5), then processes each identified multimedia 
file (e.g. an image, audio or video file) to extract tokens (Fig. 6 and Figs. 7a- 7d), and then 
places the token with associated reference data into the index (Fig.8) to yield the index 
structure illustrated in Fig. 9. Nelson's reference data stored with each token in the 
index describes, for example, the position of the multimedia component in the Web page 
(e.g. its character position or offset) and other type-dependent information useful for 
comparing instances of the same type of multimedia component, such as color, texture, 
and the like (Nelson, col. 3, lines 30-36) 

If, as the Examiner suggests, the tokens were instead inserted into the HTML 
page as meta tags in the document header, a number of consequences would result: (a) 
the proposed modification would complicate rather than simplify Nelson's process 
because the inserted meta tags would need to be again extracted after they were inserted - 
an unnecessary step since they are already available to Nelson's system after being 
extracted from the multimedia object; (b) the addition of the meta tags, being unnecessary . 
in Nelson's system, would "bulk up" the Web pages requiring greater storage capacity 
and serving no useful purpose; (c) by placing the tokens in meta tags in the HTML 
header, the token data would no longer be located where the reference tags were located, 
and the positional searching made possible by Nelson's reference data list 906 would no 
longer work as intended. 

It is accordingly incorrect to state, as the Examiner does, that "The indexing of the 
tokens would be the same regardless of where they are stored and Nelson would 
successfully commonly index the multimedia documents and their multimedia components 
either way. " Nelson's indexing would not be the same if the tokens were stored in the 
Web pages rather than being stored with position and other data in the inverted file index 
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structure taught by Nelson and numerous query functions which are important to 
Nelson's purposes would not function if the proposed modification was made. 

In short, one skilled in the art would have no good reason to make the 
modification the Examiner suggests, and would have several very good reasons NOT to 
make the modification the Examiner suggests. 

The Examiner's Answer did not address the facts 

advanced in the Appeal Brief that demonstrate non-obviousness 

In Section 10 of the Examiner's, the Examiner merely restated his contentions, 
previously expressed, that claimed elements of appellants' invention are "suggested by" 
Nelson. But the Examiner has cited no teaching in Nelson to support of this contention; 
moreover, the Examiner has not addressed or attempted to refute the facts set forth in 
appellants* Appeal Brief that demonstrate that one skilled in the art would not be 
motivated to modify the Nelson system to yield the invention claimed instead of the 
system Nelson teaches. 

The Examiner repeated the earlier contention that one skilled in the art would be 
motivated to modify Nelson's system in order to "simplify the software extraction," but 
the Examiner did not respond to the explanation on page 7 of the Appeal Brief that the 
proposed modification would complicate not simplify the Nelson's system. 

The Examiner did not dispute that Nelson "teaches away" from using 
conventional text based indexing of Web pages which incorporate multimedia data as 
explained at page 8 of the Appeal Brief. 

The Examiner did not dispute that the HTML specification nowhere suggests that 
meta tags could or should be used to describe a Web page's multimedia components as 
explained on page 8 of the Appeal Brief. 

The Examiner did not dispute that, if metadata describing referenced multimedia 
content was placed in original Web page instead of being placed in an inverted file 
multimedia index as Nelson teaches, that modification would render Nelson's system 
unsatisfactory for its intended purpose as explained on pages 8-9 of the Appeal Brief. 

Nor did the Examiner dispute that no useful purpose would be served by 
combining the multimedia data with the original Web page in addition to also placing that 
data into Nelson's inverted tile index, since Nelson's inverted file index system can 
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already perform the needed search functions and rewriting Web pages to include 
metadata would consume storage space without serving a useful purpose, as explained on 
page 9 of the Appeal Brief. 

The Examiner has thus relied solely on his ipse dixit assertion that Nelson 
"suggests" these claimed elements of appellants' invention, when in fact Nelson plainly 
does not make any such suggestion, and indeed teaches a different solution that one 
skilled in the art would have no reason whatsoever to modify. 

In this regard, it is submitted that Nelson faced the same problem but taught a 
different solution. There nothing in Nelson's teaching that would motivate one skilled in 
the art to depart from Nelson's teaching and instead do something entirely different; 
namely, process the Web pages and the referenced external multimedia objects to 
automatically produce enhanced Web pages so that conventional search engines, not a 
special purpose search engine of the type taught by Nelson, could be employed. Nelson 
teaches replacing conventional text based indexing and search facilities with facilities 
that are capable of indexing and searching both the text content of Web pages and the 
content of the multimedia data those pages refer to. Appellants' system uses existing 
search engines that are in place and already in use. Nelson teaches that such existing 
search engines have shortcomings and should be modified to be capable of indexing 
compound documents. Appellants' invention allows existing search engines to index and 
search for referenced multimedia data without requiring that those search engines be 
modified. Appellants 1 invention allows publishers of Web based multimedia content to 
process and enhanced that content so that it becomes more accessible without requiring 
the modifications to existing search engines that Nelson advocates. Nothing in Nelson 
suggests that that. Nothing in the HTML specification suggests that. Nothing but 
appellants' disclosure suggests that. 

The Examiner's Answer also did not address the law cited 
in the Appeal Brief that demonstrate non-obviousness 

Not only has the examiner ignored the facts, he has ignored the law. He has 

maintained his contention that the claimed invention would have been obvious in view of 

Nelson even though the only teaching or suggestion to combine extracted annotations 

describing referenced multimedia data with the Web page to form an enhanced web page 
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is found, not in Nelson, but in appellant's disclosure. The Examiner has thus failed to 
establish a prima facie case of obviousness as required by In re Vaeck 7 947 F.2d 488, 20 
USPQ2d 1438 (Fed. Cir. 1991) and M.P.E.P. §2143. 

Having identified no teaching of the claimed combination in Nelson, the 
Examiner relies on "the level of ordinary skill in the art" to provide that suggestion, 
contrary to Al-Site Corp. v. VSI InVl Inc., 174 F.3d 1308, 50 USPQ2d 1161 (Fed. Cir, 
1999). 

The Examiner's reliance on the HTML specification as establishing the level of 
ordinary skill is misplaced. That specification permits annotations to be inserted into a 
Web page as meta tags, but nowhere suggests that these meta tags could be or should be 
generated by processing the content of multimedia data referred to in the Web page. The 
Examiner thus ignores the well established, common-sense principle that the mere fact 
that references can be combined or modified does not render the resultant combination 
obvious unless the prior art also suggests the desirability of the combination. In re Mills, 
916 F.2d 680, 16 USPQ2d 1430 (Fed. Cir. 1990) * * * Although a prior art device "may 
be capable of being modified to run the way the apparatus is claimed, there must be a 
suggestion or motivation in the reference to do so." 916 F.2d at 682, 16 USPQ2d at 
1432.). See also In re Fritch, 972 F.2d 1260, 23 USPQ2d 1780 (Fed. Cir. 1992). 

Finally, the Examiner ignores the well established law that where, as here, the 
proposed modification would render the system being modified unsatisfactory for its 
intended purpose, it cannot be said that references provide a suggestion or motivation to v 
make the proposed modification. In re Gordon, 733 F.2d 900, 221 USPQ 1125 (Fed. Cir. 
1984) and M.P.E.P. §2143.01. 

The Rejection of Claim 6 

The Examiner concedes that Nelson does not teach combining the Web page with 
metadata describing a referenced multimedia resource where that metadata is obtained 
from the operating system that uses to access that resource as set forth in dependent claim 
6 The Examiner contends that, because Nelson uses the operating system to access and 
process files that Nelson "recognizes the benefit of collecting information from the 
operating system.' 7 But those observations do not justify the conclusion to which the 
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Examiner then leaps: that "Nelson therefore suggests that the this (sic) metadata would 
be useful as actual information in character-based tokens ..." While it is true, as 
appellants' recognized, that information from the operating system, such file system time- 
stamp information indicating when referenced media data files were initially created and 
last modified, can be useful in an indexing system, there is no evidence whatsoever that 
Nelson recognized that fact, or converted any such information from the operating system 
into tokens, and certainly no evidence that Nelson generated an enhanced Web page 
containing such information. The Examiner's rejection of dependent claim 6 is 
completely unsupported by the facts. 

Conclusion 

The Examiner has failed to establish a prima facia case of obviousness. Nelson, 
the single prior art reference relied upon, fails to disclose the claimed invention, and the 
Examiner has cited nothing, other than his own argument, to substantiate his contention 
that Nelson somehow "suggests" the claimed invention. The facts, which the Examiner 
has ignored, demonstrate the opposite: Nelson teaches away from the claimed 
combination and one skilled in the art would not be motivated to modify Nelson as the 
Examiner proposes. 

The final rejection of claims 1-17 should be reversed. 



Respectfully submitted, 




Dated: February 6, 2006 



Charles G. Call, Reg. 20,406 



9 



PACE 10/11 * RCVD AT 2/6/2006 12:53:40 PM [Eastern Standard Time] * SVR:USPTO-EFXRF-6/35 * DNIS:2738300* CSID: 1-508-629-6540 CCALL * DURATION (mm-ss):06-48 



To: U.S. Patant'and* Trademark Office Page 1 1 of 1 1 



2008-02-06 17:53:58 (GMT) 



1 -508-629-6540 CCALL From : Charles Call 



Certificate of Transmission under 37 CFR 1.8 



I hereby certify that this Appellant 's Reply Brief is being transmitted by facsimile to the 
central facsimile number of the U.S. Patent and Trademark Office, (571) 273-8300, on February 
6, 2006. 



Dated: February 6, 2006 



Signature 




Charles G. Call, Reg. No. 20,406 

USPTO Customer No. 021253 

1161 Marlin Court 

Marco Island, FL 34145 

Ph. (617) 820-5227 - Fax (508) 629-6540 

cal l@pat entsoft com 



10 



PAGE 11/11 * RCVD AT 2/6/2006 12:53:40 PM [Eastern Standard Time] * SVR:USPTO-EFXRF-6/35 * DNlS:2738300 * CSID: 1-508-629-6540 CCALL * DURATION (mm-ss):06-48 



